probabilistic symbol-refined tree substitution grammar
Statistical Parsing with Probabilistic Symbol-Refined Tree Substitution Grammars
Shindo, Hiroyuki (NTT Communication Science Laboratories) | Miyao, Yusuke (National Institute of Informatics) | Fujino, Akinori (NTT Communication Science Laboratories) | Nagata, Masaaki (NTT Communication Science Laboratories)
We present probabilistic Symbol-Refined Tree Substitution Grammars (SR-TSG) for statistical parsing of natural language sentences. An SR-TSG is an extension of the conventional TSG model where each nonterminal symbol can be refined (subcategorized) to fit the training data. Our probabilistic model is consistent based on the hierarchical Pitman-Yor Process to encode backoff smoothing from a fine-grained SR-TSG to simpler CFG rules, thus all grammar rules can be learned from training data in a fully automatic fashion. Our SR-TSG parser achieves the state-of-the-art performance on the Wall Street Journal (WSJ) English Penn Treebank data.